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Relating ACT Writing Scores to First-Year 
English Composition Grades 


Justine Radunzel, PhD 


Introduction 


The optional ACT® writing test is designed 
to measure students’ writing skills — 
specifically, those skills emphasized and 
acquired in high school English classes and 
important for success in entry-level college 
composition courses. The test was first 
introduced in 2005, and in fall 2015, a 
number of enhancements to the former 
version were introduced. The 
enhancements included redesigning the 
writing prompts, extending the testing time 
from 30 minutes to 40 minutes, and using 
an analytical rubric on four writing domains 
for scoring instead of using a holistic six- 
point rubric. For the latter enhancement, the 
two independent trained readers assigned 
to the essay now use a six-point scoring 
rubric for each of the four domains 
evaluated; the domains include Ideas and 
Analysis, Development Support, 
Organization, and Language Use and 
Conventions (ACT, 2019; pp. 3.15 — 3.22)." 
Each writing domain score is calculated as 
the sum of the two readers’ ratings for the 
domain with the rounded average of the four 
domain scores being reported as the 
enhanced ACT writing score. Information 
from the enhanced ACT writing test aims to 
inform postsecondary institutions about 
students’ ability to think critically about an 
issue, consider different perspectives on it, 
and compose an effective argumentative 
essay in a timed condition. 


By design, one purpose of the essay is to 
identify students’ strengths and weaknesses 
in writing so that they can skill up in areas in 
need of improvement while there is still time 
to do so in high school. Another purpose is 
to help postsecondary institutions inform 
college course placement decisions for 
writing-intensive courses and identify 
students placed into a standard-level 
English Composition course that may 
benefit from additional academic supports 
and services. Empirical evidence exists 
supporting the use of the former ACT writing 
test, as well as showing the incremental 
value of using ACT writing scores with ACT 
English scores, for these purposes (ACT, 
2009). Given the recent enhancements to 
the ACT writing test, there is a need to 
revisit these topics. 


With this background in mind, the objective 
of the current study was to examine the 
validity of using the enhanced ACT writing 
test for predicting grades in a first-year 
college English Composition course, alone 
and in combination with other measures 
including the ACT English score and high 
school grade point average (HSGPA). Two 
types of evidence were evaluated: (a) 
statistical measures of the overall strength 
of the relationship between the predictors 
and course grades and (b) decision-based 
statistics such as accuracy and success 
rates for identifying students likely to 
achieve a specific grade or higher in the 


ACT 


00006 


ACT.org/research 
© 2019 by ACT, Inc. All rights reserved. | R1789 


ACT Research & Policy | Technical Brief | November 2019 2 


course. The following questions were 
addressed in this study: 


1. Do students with higher ACT writing 
scores perform better in a first-year 
college English Composition course? 


2. Does the ACT writing score enhance 
the prediction of grades earned in a 
first-year college English Composition 
course, above ACT English score and 
HSGPA? 


Data and Methods 
Study Sample 


The study sample was comprised of 6,477 
students from the 2016 freshman cohort 
who took the enhanced ACT writing test 
during the 2015-2016 academic year as a 
senior in high school and a first-year credit- 
bearing English Composition | course during 
their freshman year in college. Students 
were also required to have an ACT English 
score and a HSGPA.? Students were from 
28 different postsecondary institutions that 
had participated in research services 
offered by ACT and had a minimum of 35 
students meeting the study inclusion 
criteria. All but one of the institutions was a 
four-year institution. Three-fourths were 
public institutions, nearly two-thirds (64%) 
had total enrollments of 10,000 or more, 
86% had selective or traditional admissions 
policies, and 89% were from the Midwest 
and Southern census regions.° 


Female students made up more than one- 
half (57%) of the sample, and nearly one- 
fourth (23%) were racial/ethnic minority 
students, primarily African American (15%) 
and Hispanic (8%). Students from other 
underserved learner groups were also 
represented in the sample. Of those 


providing the corresponding information, 
10% indicated that their parents had no 
college experience, 18% indicated that their 
annual family income was less than 
$36,000, and 5% indicated that English was 
not the most frequently spoken language at 
home.* 


Measures 


Outcomes. The course grades earned in 
the English Composition course were 
provided by institutions under a plus-minus 
grading system and were coded as 0.0 (for 
an F) through 4.0 (for an A), or as a 
withdrawal. In some analyses, the course 
grade outcome was treated as a continuous 
outcome. In other analyses, course grades 
were dichotomized to examine course 
success rates using three different criteria 
levels that included earning an A grade, a B 
or higher grade, or a C or higher grade. The 
19 students with a withdrawal grade were 
treated as an unsuccessful event in the 
course success rate analyses, but they 
were not included in the analyses that 
examined course grades as a continuous 
outcome. 


Predictors. The predictors included in this 
study were students’ ACT writing scores, 
ACT English scores, and self-reported 
HSGPAs. The ACT writing score was the 
rounded average of the four ACT writing 
domain scores and ranged from 2 to 12. 
The ACT English score ranged from 1 to 36 
and represented the score obtained from 
the multiple-choice ACT English test. 
HSGPA was based on students’ self-reports 
of their coursework taken in up to 23 
specific courses in English, mathematics, 
social studies, and science, and the grades 
earned in those courses. Prior studies have 
shown that students report high school 
coursework and grades accurately relative 
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to information provided in their transcripts 
(Sanchez & Buddin, 2016). 


Analyses were also conducted using 
English HSGPA, as well as scores from the 
ACT writing test that are placed on a 1 to 36 
scale, equated across test forms, and 
computed for the purpose of contributing to 
students’ ACT English Language Arts (ELA) 
scores (ACT, 2019, pp. 9.3-9.4).° Results 
based on these alternative predictors were 
not provided in this report as they were 
found to be comparable to those reported 
for overall HSGPA and for the 2 to 12 ACT 
writing score currently provided to students 
and institutions on ACT score reports or in 
data files. 


Analyses 


Because students were nested within 
institutions, hierarchical linear regression 
models were used to relate the predictors 
(ACT scores and HSGPA) to English 
Composition course grades, and 
hierarchical logistic regression models were 
used for the various dichotomous success 
criteria. Given the variation that can exist in 
course content and grading standards 
across institutions, the intercept and slopes 
of the predictors were allowed to vary 
across institutions. The predictors included 
in the models were standardized to have a 
mean of zero and a standard deviation of 
one according to the overall statistics of the 
primary study sample. The standardized 
regression coefficients for each predictor 
are reported across the seven models 
estimated and provide information about the 
relative strength of the predictors in relation 
to the outcome. Model 1 included ACT 
writing score, model 2 included ACT English 
score, model 3 included HSGPA, model 4 
included ACT writing and English scores, 
model 5 included ACT writing score and 


HSGPA, model 6 included ACT English 
score and HSGPA, and model 7 included all 
three predictors. 


To further examine the predictive and 
incremental validity of using the ACT writing 
score to estimate students’ English 
Composition course grade, the multiple 
correlation (R) was computed. The multiple 
R provides a measure of the strength of the 
relationship between the predictors and the 
continuous course grade outcome. It ranges 
between 0 and 1. A higher value indicates 
better prediction. Multiple R values were 
computed for each institution. 


Because ACT scores and HSGPA are often 
used to help select students who are 
academically ready for college courses, 
multiple R values that corrected for range 
restriction in the variances of the predictors 
were also computed. The Pearson-Lawley’s 
multivariate correction procedure described 
by Sackett and Yang (2000) was utilized to 
obtain these corrected values. Institution- 
specific applicant pools were approximated 
and utilized to estimate the unrestricted 
population variance-covariance matrix of the 
predictors. The applicant pool for an 
institution was comprised of enrolled 
students as well as non-enrolled students 
who sent scores to the institution during the 
same time period as the enrolled cohort; all 
students included in the applicant pools had 
taken the optional enhanced ACT writing 
test, the ACT English test, and had a 
HSGPA value available. The total applicant 
pool for this study was comprised of 96,197 
students, and some students belonged to 
more than one applicant pool as they had 
sent scores to more than one institution 
included in this study.° The multivariate 
correction procedure was conducted using 
the mv.correction function within the R 
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Selection package (Birnbaum, Paulson, & 
Andrews, 1950). 


To further examine the predictive and 
incremental validity of using the ACT writing 
score to estimate a student’s chances of 
earning a specific grade or higher in their 
English Composition course, the following 
four validity measures were computed: 
logistic R, maximum accuracy rate (AR), the 
increase in maximum accuracy rate (AAR), 
and maximum success rate (SR). Logistic R 
is the standard deviation of the estimated 
logit function and it measures the overall 
predictive strength of the logistic model 
(Allen & Le, 2008). The higher the logistic R, 
the stronger the relationship is between the 
predictors and chances of earning the 
specific grade or higher in the course. The 
logistic R was estimated for each institution 
using the institution’s applicant pool. 


Based on the predictors for each student, a 
probability of earning a specific grade or 
higher in the course at the institution 
attended can be estimated from the logistic 
regression model. These estimated 
probabilities might be used to identify 
students who could benefit from additional 
academic supports to achieve a specific 
grade or higher in the course. For example, 
students with an estimated probability below 
0.50 might be identified as those who could 
benefit from additional academic supports to 
achieve the specific outcome in the course 
(e.g., a B or higher grade). 


The AR represents the percentage of 
correct classifications and is calculated as 
the sum of (a) the percentage of students 
who will earn the specific grade or higher 
and are predicted to achieve the specific 
outcome and (b) the percentage of students 
who will not obtain the specific grade and 
are predicted to not achieve the specific 


grade. It can be shown that if the probability 
function crosses 0.50 within the observed 
range of the predictor, then the AR is 
maximized when a 0.50 probability is used 
for making the classifications (referred to as 
the maximum AR; Sawyer, 1996), that is, 
when students with a 0.50 or higher 
probability are classified as likely to achieve 
the specific grade outcome, and those with 
a probability lower than 0.50 are classified 
as not likely to achieve the specific grade 
outcome. 


The corresponding maximum AAR is the 
increase in the percentages of correct 
classifications when the prediction model is 
used as compared to expecting all students 
to achieve the specific outcome, and the 
corresponding maximum SR is an estimate 
of the percentage of students who achieve 
the specific grade criteria among those 
expected to achieve the outcome. The 
maximum AR, AAR, and SR were estimated 
for an individual institution only in cases 
when the institution’s probability function 
crossed 0.50 (referred to as having a 
“viable” probability model for which the 
maximum AR can be estimated). Similar to 
the calculation of the corrected correlation 
coefficients and logistic R values, these 
statistics were estimated for the institution’s 
applicant pool. For more details on the 
theoretical basis of the maximum AR, AAR, 
and SR and how they are computed, the 
reader is referred to pages 11.39 to 11.42 of 
the ACT Technical Manual (ACT, 2019). 


To aggregate and summarize over the 
institution-specific results, the median (i.e., 
the 50th percentile), the 25th percentile, and 
the 75th percentile for each statistical 
measure examined in this study (e.g., 
multiple R, logistic R, AR) were computed to 
get a sense of the typical value and the 
range of the middle 50% of values across 
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institutions. Both unweighted and weighted 
analyses were used to summarize across 
institutions. In the unweighted analyses, 
each institution contributed equally to the 
results. In the weighted analyses where 
weights were assigned as the number of 
students at the institution divided by the 
total number of students in the analyses, 
institutions with a larger number of students 
contributed more to the results. 


Results 


Descriptive Statistics of Sample 
and Applicant Pool 


Table 1 provides descriptive statistics on the 
predictors and the course grade outcome 
summarized across institutions for the 
course sample and for the applicant pool 
sample. For each predictor, the typical 
average value was somewhat comparable 


between the two samples. The typical 
standard deviation across institutions was 
slightly smaller in the course sample than in 
the applicant pool (1.4 vs. 1.5 for ACT 
writing score, 4.3 vs. 5.5 for ACT English 
score, and 0.45 vs. 0.50 for HSGPA). The 
three predictors were moderately correlated 
with one another in the applicant pool, 
ranging from .312 between ACT writing 
score and HSGPA to .490 between ACT 
writing and ACT English scores. The typical 
correlations between the predictors were 
lower in the course sample. The typical 
percentage of students earning a B or 
higher grade in the English Composition 
course was 75%, but this percentage did 
vary across institutions (from the lower 
quartile [Q1] of 67% to the upper quartile 
[Q3] of 81%). A majority of the students 
earned a C or higher grade in the course 
(median percentage = 91% across 
institutions). 


Table 1. Descriptive Statistics on the Predictor and Course Grade Information Summarized across 


Institutions by Sample 


eda :\o |(oa ko) mr-lale Med e-(o(-Mlalie) Q1 Median Q3 Q1 Median Q3 
6.4 6.8 7.2 6.4 6.8 7.1 


Average ACTW 
Average ACTE 19.7 
Average HSGPA 3.25 
Correlations between predictors 
ACTW and ACTE 271 
ACTW and HSGPA .093 
ACTE and HSGPA .149 
Average course grade 2.82 
% earning A 19 
% earning B or higher 67 
% earning C or higher 87 
Sample size 50 


21.7 
3.34 


341 
.160 
243 
3.04 
33 
15 
91 
121 


23.4 19.7 21.5 23.6 
3.44 3.31 3.40 3.48 
.390 452 490 O11 
.262 277 312 337 
365 401 440 466 
3.21 

46 

81 

94 
321 422 2,130 6,125 


Note. Based on 28 participating institutions. Q1 = 25th percentile, and Q3 = 75th percentile. ACT writing score 
ranged from 2 to 12, and ACT English score ranged from 1 to 36. ACTW = ACT writing score. ACTE = ACT English 


score. HSGPA = high school grade point average. 
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Relationship with Course Grade Even when one or both of the other two 


ae a predictors were added to the model, the 
As illustrated in Figure 1, the ACT writing ACT writing score was found to be a 
score was positively related to the grade significant predictor of the grade students 


students eamed in their English eared in their English Composition course 
Composition course. The estimated course (see Table 2; p value < 0.0001).The 


grade for students with an ACT writing 


standardized coefficient for ACT writing 
score of 4 was 2.66 (around a C+) ata 


a, ee score from the hierarchical linear models 
ty pies b Suton. In.compal sch sttaelts was similar to or slightly higher than that for 


with an ACT writing score of 7, 9, and 12 ACT English score and lower than that for 
were estimated to earn a grade of 3.09 HSGPA. 


(around a B), 3.37 (around a B+), and 3.79 
(between an A- and A), respectively. 


Figure 1. Estimated English Composition Course Grade as a Function of ACT Writing Score at a Typical 
Institution 
4.00 
3.50 
3.00 
2.50 
2.00 
1.50 
1.00 
0.50 
0.00 


Estimated course grade 


2 3 4 5 6 7 8 9 10 11 12 


ACT writing score 


Table 2. Standardized Linear Regression Coefficients across the Models 


Estimate (standard error) 


Model Intercept ACT writing ACT English 


1 3.114 (0.048) 0.211 (0.028) 

2 3.119 (0.050) 0.193 (0.023) 

3 3.123 (0.047) 0.289 (0.026) 
4 3.150 (0.046) 0.164 (0.026) 0.138 (0.021) 

5 3.165 (0.042) 0.151 (0.022) 0.261 (0.024) 
6 3.162 (0.044) 0.119 (0.018) 0.260 (0.025) 
7 3.182 (0.041) 0.127 (0.021) 0.080 (0.016) 0.247 (0.024) 


Note. Based on 28 participating institutions. p values for the intercept and all predictors in each model were 
<0.0001. The predictors were grand-mean centered and standardized to have a mean of 0 and standard deviation of 
1 according to the following grand means (standard deviations) from the course sample: 7.2 (1.5) for ACT writing, 
23.5 (5.3) for ACT English, and 3.47 (0.44) for HSGPA. Estimated variances across institutions of standardized linear 
regression coefficients are provided in Table A1 of the Appendix. HSGPA = high school grade point average. 
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To further illustrate the incremental value of across institutions in the course sample 
adding ACT writing scores to the prediction increased from .324 to .349 in unweighted 
model, Table 3 provides the typical analyses and from .307 to .342 in weighted 
uncorrected and corrected multiple analyses. When corrections for range 
correlations across institutions associated restriction in the predictors were applied 
with the various prediction models based on using the applicant pool sample, the 
unweighted and weighted analyses. When corresponding typical multiple correlation 
ACT writing score was added to the model across institutions increased from .412 to 
that included ACT English score and .433 in unweighted analyses and from .418 
HSGPA, the typical multiple correlation to .424 in weighted analyses. 


Table 3. Multiple Correlations (Uncorrected and [Corrected]) across Institutions for Predicting English 
Composition Course Grade 


Unweighted analyses Weighted analyses 
led «Xe | (ea Ke) x) 


Q1 Median Q3 Q1 Median Q3 
Single-predictor models 
1 ACTW 107 (185)  .179(.260) .287(.336) .104(.140) .167(.246) —.206 (.296) 
Za ACTE 418 (211) 179.301) 237 (.347) 121.207) 178 (.306) —_.214 (.328) 
3. HSGPA 198 (.256)  .267(.358) .385(.433) 197.262) .271(.361) _.311 (.428) 
Multiple-predictor models 
ACTW + 
OO cre 471 (.278) 245 (.333) .380(.398) .142(.208) 213.319) _.242 (.366) 
ACTW + eae ae) 
in 255 (.302) .331(.414) .451(.507) 207.267) 327.412) .339(.457) 
6 ch 219(.289)  .324(.412) .417(.470) .214(.284) .307(.418) _.341 (.451) 
ACTW + 
7 ACTE+ 297 (.329)  .349(.433) 458 (.513) .224(.293) 342.424) 354 (.479) 
HSGPA 


Note. Based on 28 participating institutions. Q1 = 25th percentile, and Q3 = 75th percentile. ACT writing score 
ranged from 2 to 12, and ACT English score ranged from 1 to 36. In the weighted analyses, institutions with larger 
numbers of students were assigned larger weights (i.e., number of students at institution divided by total number of 
students in analysis). Corrected correlations shown in parentheses were corrected for multivariate range restriction 
using the institution’s applicant pool as the unrestricted population. ACTW = ACT writing score. ACTE = ACT English 
score. HSGPA = high school grade point average. 
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Relationship with a Student’s 
Chances of Earning a Specific 
Grade or Higher 


Regression models and coefficients. 
As illustrated in Figure 2, ACT writing score 
was positively related to a student’s 
chances of earning a B or higher grade in 
their college English Composition course. 
The chances increased from 44% for those 
with an ACT writing score of 2 to 93% for 
those with a score of 12. Results for earning 
an A grade and for earning a C or higher 
grade are also provided in the figure. 


When the ACT writing score was used 
alone or in combination with the other two 
predictors to estimate students’ chances of 
earning a specific grade or higher in a 


college English Composition course, the 
standardized logistic regression coefficient 
for ACT writing score was significantly 
different from zero (see Table 4; each p < 
0.01 across models and outcomes). Based 
on the model that included all three 
predictors (labeled Model 7), the 
standardized coefficient was smaller for 
ACT writing score than for ACT English 
score for the A outcome, but it was more 
comparable between the two predictors for 
the B or higher outcome and it was larger 
for the C or higher outcome. For this latter 
outcome, the ACT English score was not a 
significant predictor when one of the other 
predictors was included in the model. This 
result is likely due to the relatively high 
percentage of students earning a C or 
higher grade in English Composition and 
the predictors being moderately correlated. 


Figure 2. The Probability of Success in a College English Composition Course by ACT Writing Score ata 
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HSGPA did not change much across the 
criterion levels, especially for HSGPA. 
Having said that, the standardized 
coefficients for both ACT writing and ACT 
English scores were smaller than that for 
HSGPA across all models and outcomes. 


Interestingly, as the criterion level increased 
(from a C or higher to an A) so did the 
predictive strength of ACT English score (as 
measured by the standardized logistic 
regression coefficient). In comparison, the 
predictive strength of ACT writing score and 


Table 4. Standardized Logistic Regression Coefficients across Logistic Regression Models of Earning a 


Specific Grade or Higher in English Composition 


Estimate (standard error) 


Intercept ACT writing ACT English HSGPA 
A grade 

1 -0.681 (0.147) 0.392 (0.049) 

2 -0.603 (0.154) 0.516 (0.059) 

3 -0.665 (0.152) 0.587 (0.036) 
4 -0.581 (0.158) 0.263 (0.043) 0.427 (0.061) 

5 -0.614 (0.159) 0.323 (0.046) 0.549 (0.037) 
6 -0.565 (0.164) 0.393 (0.058) 0.510 (0.038) 
7 -0.548 (0.168) 0.235 (0.042) 0.317 (0.060) 0.497 (0.038) 

B or higher grade 

1 1.233 (0.107) 0.430 (0.059) 

2 1.268 (0.105) 0.460 (0.055) 

3 1.306 (0.107) 0.596 (0.058) 
4 1.329 (0.101) 0.324 (0.058) 0.346 (0.050) 

5 1.393 (0.102) 0.332 (0.052) 0.548 (0.056) 
6 1.407 (0.102) 0.324 (0.051) 0.534 (0.058) 
7 1.447 (0.103) 0.269 (0.053) 0.233 (0.049) 0.512 (0.057) 

C or higher grade 

1 2.643 (0.172) 0.380 (0.076) 

2 2.596 (0.176) 0.252 (0.085) 

3 2.769 (0.169) 0.574 (0.053) 
4 2.675 (0.169) 0.334 (0.078) 0.133 (0.089)* 

5 2.853 (0.164) 0.274 (0.071) 0.536 (0.054) 
6 2.793 (0.166) 0.088 (0.092)* 0.558 (0.055) 
7 2.842 (0.162) 0.267 (0.073) -0.006 (0.095)* 0.537 (0.055) 


Note. Unless denoted with *, p values for the intercept and all predictors in each model were < 0.01. * means that p 
value > 0.05. The predictors were grand-mean centered and standardized to have a mean of 0 and standard 
deviation of 1 according to the following grand means (standard deviations) from the course sample: 7.2 (1.5) for ACT 
writing, 23.5 (5.3) for ACT English, and 3.47 (0.44) for HSGPA. Estimated variances across institutions of 
standardized logistic regression coefficients are provided in Table A1. HSGPA = high school grade point average. 


ACT Research & Policy | Technical Brief | November 2019 5 


To help assess the fit of the logistic Figure 3 further illustrates that the ACT 
regression models and the incremental writing score helps to differentiate the 
benefit of using multiple measures to chances of success among students with 
estimate a student’s chances of success, the same ACT English score and HSGPA. 
Table 5 provides results on the logistic R Specifically, the example provides the 
across the models for the B or higher chances of earning a specific grade or 
outcome. higher in English Composition as a function 

of ACT writing score for students with an 
The logistic R or predictive strength ACT English score of 24 and a HSGPA of 
measure was highest when all three 3.47 (the approximate course sample mean 
predictors were included in the model. This values). The conditional probabilities of 
result was seen for both the unweighted and success for earning a B or higher grade 
weighted analyses and for the A grade range from 0.63 to 0.91 across the Writing 
outcome (results not shown). scale from 2 to 12. 


Table 5. Overall Predictive Strength, Logistic R across Institutions for Logistic Regression B or Higher 
Course Grade Models 


Unweighted analyses Weighted analyses 
Predictors Q1 Median ek) eh Median ek) 


Single-predictor models 


1 ACTW 0.339 0.396 0.456 0.335 0.349 0.403 
ACTE 0.318 0.362 0.446 0.310 0.348 0.382 

3 HSGPA 0.463 0.588 0.711 0.407 0.445 0.584 

Multiple-predictor models 

ACTW + 

4 ACTE 0.418 0.463 0.549 0.366 0.441 0.505 
ACTW + 

5 HSGPA 0.566 0.652 0.792 0.523 0.569 0.657 
ACTE + 

6 HSGPA 0.527 0.635 0.774 0.487 0.511 0.641 
ACTW + 

7 ACTE + 0.590 0.680 0.837 0.533 0.592 0.686 
HSGPA 


Note. Based on 28 participating institutions. Q1 = 25th percentile, and Q3 = 75th percentile. ACT writing score 
ranged from 2 to 12, and ACT English score ranged from 1 to 36. In the weighted analyses, institutions with larger 
numbers of students were assigned larger weights (i.e., number of students at institution divided by total number of 
students in analysis). ACTW = ACT writing score. ACTE = ACT English score. HSGPA = high school grade point 
average. The logistic R measures the overall predictive strength of the model and is defined as the standard deviation 
of the estimated logit function. The higher the logistic R, the stronger the relationship is between the predictors and 
the chances of earning the specific grade or higher in the course. 
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Figure 3. The Conditional Probability of Success in a College English Composition Course by ACT 
Writing Score at a Typical Institution for Students with an ACT English Score of 24 and a HSGPA of 3.47 
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Decision-based statistics. Table 6 
provides the typical maximum AR, AAR, 
and SR for the single-predictor models for 
the A and B or higher outcomes. The lower 
and upper quartiles (from Q1 to Q3) for 
these measures are provided in Table A2 in 
the Appendix. Results are provided for the 
16 institutions that had viable models (i.e., 
the probability functions crossed 0.50) for all 
three predictors. Results are not reported 
for the C or higher outcome because of the 
small number of viable models.’ 


For the A outcome, the typical maximum AR 
and SR were lower for the ACT writing 
score than for the other two predictors; 
these measures were more comparable 
between ACT English score and HSGPA. 

In comparison, for the other outcome of 


earning a B or higher grade, the typical 
maximum AR and SR for the ACT writing 
score found in this study were similar to 
those for the ACT English score from this 
study and in another earlier study of 215 
postsecondary institutions (ACT, 2019, 
Table 11.25). For this same outcome, the 
typical AAR for the ACT writing and English 
scores was relatively small (less than 1 
percentage point) and was somewhat 
smaller than that reported for HSGPA in this 
study (about 3 percentage points) and for 
ACT English score from an earlier study 
(ACT, 2019, Table 11.25; 2 percentage 
points). For each predictor, the typical AAR 
associated with using the predictors was 
higher for the A outcome than it was for the 
B or higher outcome. 
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Table 6. Median Maximum Accuracy Rates (AR), Increase in Accuracy Rates (AAR), and Success Rates 
(SR) across Institutions for Single-Predictor Logistic Regression Course Grade Models 


< Unweighted analyses Weighted analyses 
ed K:Xe| (ei ke) a) 
AR AAR SR AR AAR SR 


A grade 
1 ACTW 59.4 15.5 56.6 59.3 12.1 55.5 
2 ACTE 62.3 18.0 58.4 62.1 15.1 59.5 
3 HSGPA 63.2 17.4 58.6 63.3 16.8 58.3 
Bor higher grade 
1 ACTW 66.9 0.5 67.6 68.5 0.1 68.7 
2 ACTE 66.9 0.3 67.5 68.2 0.2 68.7 
3 HSGPA 71.1 2.8 71.8 70.1 1.3 71.3 


Note. Based on 16 out of the 28 participating institutions that had viable models (i.e., probability curves that crossed 
0.50) for all three predictors. In the weighted analyses, institutions with larger numbers of students were assigned 
larger weights (i.e., number of students at institution divided by total number of students in analysis). ACTW = ACT 
writing score. ACTE = ACT English score. HSGPA = high school grade point average. The corresponding 
interquartile ranges (from Q1 to Q3) for these measures are provided in Table A2 in the Appendix. 


Table 7 provides the typical decision-based 
statistics for two of the multiple-predictor 
models that allow for the examination of the 
incremental value associated with adding 
ACT writing scores to the model that 
includes ACT English scores and HSGPA. 
These two models were focused on 
because they were associated with the 
largest number of institutions with viable 
models (24 for the A outcome and 26 for the 
B or higher outcome). For both outcomes, 
there were small incremental increases of 
nearly 1 percentage point in the typical 
maximum AR, AAR, and SR. 


These findings are consistent with other 
studies that have examined the incremental 
change in the typical maximum AR when a 
cognitive measure is added to a model that 
already includes another cognitive measure 
for predicting students’ chances of earning a 
B or higher grade in English Composition 
(ACT, 2019, Table 11.26; Westrick & Allen, 
2014, Tables 7-9). In general, English 
courses tend to have higher percentages of 
students earning a B or higher grade, 
leading to smaller increases in accuracy 
rates. This phenomenon occurs regardless 
of the predictor being used. 
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Table 7. Median Maximum Accuracy Rates (AR), Increase in Accuracy Rates (AAR), and Success Rates 
(SR) across Institutions for Multiple-Predictor Logistic Regression Course Grade Models 


Unweighted analyses Weighted analyses 
Predictors AR PAVAN SS | AR AAR SR | AR rAVAN SR 


ee grade 

ACTE + 

6 _HSGPA 68.4 24.7 58.9 67.1 22.9 60.5 
ACTW + 

7 ACTE + 68.9 25.9 59.8 67.7 23.9 61.3 
HSGPA 

B or higher grade 

ACTE + 

6 HSGPA 76.3 2.2 77.7 78.4 1.3 79.4 
ACTW + 

7 ACTE + 76.7 3.1 77.8 79.1 1.6 80.2 
HSGPA 


Note. Based on 24 (26) of the 28 participating institutions that had viable A (B or higher) grade models (i.e., 
probability curves that crossed 0.50) for the two models being compared in the table. In the weighted analyses 
institutions with larger numbers of students were assigned larger weights (i.e., number of students at institution 
divided by total number of students in analysis). ACTW = ACT writing score. ACTE = ACT English score. HSGPA = 
high school grade point average. The corresponding lower and upper quartiles (25th percentile - 75th percentile) for 


these measures are provided in Table A3 in the Appendix. 


Conclusions 


The current study provides validity evidence 
for using the enhanced ACT writing score to 
measure students’ writing skills that are 
relevant for success in entry-level college 
English Composition courses. First, the 
study found that students with higher ACT 
writing scores earn higher grades on 
average than those with lower scores 
(Figure 1) and have greater chances of 
earning a specific grade (e.g., B or higher, 
or A grade) in the course (Figure 2). 
Moreover, ACT writing score was found to 
be moderately correlated with the grade 
earned in the course (Table 3) and to be 
reasonably accurate at classifying students 
likely to earn a specific grade or higher 
(Table 6). In comparison to the individual 
results for the other predictors considered in 
this study, the correlation coefficients and 


maximum ARs associated with the B or 
higher outcome for ACT writing score were 
generally similar to those for ACT English 
score, but the maximum ARs associated 
with the A outcome (excellent performance) 
were lower. This finding highlights how, in 
comparison to other predictors, the relative 
utility of a variable for accurately informing 
who is likely to succeed in the course can 
depend on the level of success being 
examined. This result was suggested by 
another study (Sawyer, 2013) that 
examined the usefulness of ACT Composite 
score and HSGPA for informing college 
admissions decisions; the study found that 
ACT Composite score performed better 
than HSGPA at greater success levels and 
worse at lower success levels. 


In the current study, the correlation 
coefficients and maximum ARs associated 
with the A outcome or the B or higher 
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outcome from the single-predictor models 
were generally slightly higher for HSGPA 
than for both ACT scores. One explanation 
for the better HSGPA results is that HSGPA 
is not only affected by content mastery, as 
is generally the case for ACT test scores, 
but it is also affected by a student’s 
academic behaviors, such as putting forth 
effort, participating in class, and completing 
homework assignments, that have also 
been found to be associated with students’ 
chances of succeeding in college (Camara, 
O’Connor, Mattern, & Hanson, 2015). 


From the multiple-predictor model results, 
ACT writing score was found to contribute 
incrementally to the prediction of the grade 
earned in English Composition, beyond 
HSGPA and ACT English score. More 
specifically, ACT writing score slightly 
improved the prediction of the grade earned 
above that when the other two predictors 
were used alone and in combination. For 
example, when ACT writing score was 
added to the models that included HSGPA 
and ACT English score, the typical 
percentage of explained variance in the 
grades earned in the course across 
institutions increased by 1 to 2 percentage 
points,® and the typical maximum AR across 
institutions associated with estimating a 
student’s chances of earning a specific 
grade or higher in the course improved by 
about 1 percentage point. We acknowledge 
that the degree of improvement associated 
with ACT writing score was small, but this 
was expected given that: (a) the three 
predictors each measure academic 
achievement and were found to be 
moderately correlated with one another, and 
(b) a very high percentage of students 
earned a B or higher grade in the English 
Composition course (typical percentage was 
75% across the institutions included in this 


study). But, as illustrated in Figure 3, small 
improvements in the multiple correlation or 
maximum ARs can be associated with 
meaningful differences in the predicted 
outcomes. Additionally, the logistic R 
measure suggested that the model that 
included all three predictors had the 
greatest predictive strength for estimating 
the likelihood of a student earning a specific 
grade or higher in the course. Taken 
together, these findings underscore the 
value of additional sources of information, 
such as ACT writing score, to gain a more 
holistic and personalized perspective of 
student readiness. 


In conclusion, results from the study 
highlight the utility of the enhanced ACT 
writing score for helping to determine a 
student’s readiness for a first-year, credit- 
bearing English Composition course. 
Postsecondary institutions and state 
agencies are encouraged to conduct local 
studies of this nature that examine the 
incremental benefit of using ACT writing 
scores to predict students’ grades and their 
chances of earning a specific grade or 
higher in first-year English Composition 
courses offered on their campuses. Such 
information can provide insights on how to 
tailor academic services and supports in this 
area to best meet the needs of incoming 
freshmen. 
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Notes 


1. If the readers’ ratings disagree by more 
than one point, a third rater evaluates 
the essay and resolves the discrepancy. 


2. Requiring students to have a HSGPA 
value reduced the sample size by 510 
students, representing only 7% of the 
initial sample. This criterion did not 
change the number of institutions 
included in the study. 


3. Characteristics for the postsecondary 
institutions were obtained from 
Integrated Postsecondary Education 
Data System (IPEDS), except for 
admissions selectivity. Admission 
selectivity was self-reported by 
institutions on the ACT Institutional Data 
Questionnaire as defined by the typical 
high school class ranks of their 
accepted freshmen: The majority of 
freshmen at highly selective schools are 
in the top 10%, selective in the top 25%, 
traditional in the top 50%, and liberal in 
the top 75% of their high school class 
(ACT, 2018). Institutions with open 
admissions policies accept all high 
school graduates to the limit of capacity. 
In terms of institutional admission 
policies, 25% were selective or highly 
selective, 61% were traditional, and 
14% were liberal or open. In terms of 
census region, 7% were from the 
Northeast, 36% from the Midwest, 54% 
from the South, and 4% from the West. 


4. Nearly all students provided their gender 
and race/ethnicity (seven missing for 
gender and two missing for 
race/ethnicity). Ninety percent of the 
students provided information about 
their parents’ education levels and the 


primary language spoken at home, and 
78% indicated their annual family 
income. 


. The comparability of the 2 to 12 writing 


test scores across forms is ensured by 
the prompt selection procedures (ACT, 
2019, chapter 2), not by an equating 
process. Beginning in fall 2016, the 1 to 
36 writing scale scores are not provided 
to students and institutions on ACT 
score reports or in data files. 


. The total number of records in the 


applicant pool was 106,428. 


. The results provided are based on the 


common set of institutions with viable 
models so that the comparisons are 
meaningful. The number of institutions 
with viable models varied by outcome 
and predictor. For the grade A outcome, 
19 institutions had viable models based 
on ACT writing scores, 22 based on 
ACT English scores, and 16 based on 
HSGPA. For the grade B or higher 
outcome, 17 institutions had viable 
models based on ACT writing scores, 18 
based on ACT English scores, and 27 
based on HSGPA. For the grade C or 
higher outcome, 1 institution had a 
viable model based on ACT writing 
scores, 0 based on ACT English scores, 
and 16 based on HSGPA. 


. The percentage of variance explained 


by the model can be found by squaring 
the multiple correlation coefficient. 
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Appendix 


Table A1. Estimated Variances Across Institutions of Standardized Linear and Logistic Regression 


Coefficients 


aL Kedex=) o) 


Variance estimate (standard error) 


ACT writing ACT English 
Grade earned (continuous scale) 


1 0.056 (0.017) 0.015 (0.007) 
2 0.061 (0.019) 0.007 (0.004)* 
3 0.052 (0.016) 0.012 (0.005) 
4 0.048 (0.015) 0.011 (0.006)* 0.004 (0.003)* 
5 0.041 (0.013) 0.007 (0.004)* 0.010 (0.005) 
6 0.044 (0.014) 0.003 (0.003)* 0.012 (0.005) 
7 0.039 (0.013) 0.006 (0.004)* 0.001 (0.002)* 0.010 (0.005) 
A grade (dichotomized) 
1 0.544 (0.158) 0.013 (0.012) 
2 0.598 (0.174) 0.037 (0.021) 
3 0.590 (0.170) -- 
4 0.625 (0.182) 0.003 (0.008)* 0.038 (0.022) 
5 0.640 (0.184) 0.010 (0.010)* -- 
6 0.680 (0.197) 0.033 (0.021)* -- 
7 0.713 (0.206) _0.003 (0.008)* 0.034 (0.022)* -- 
B or higher grade (dichotomized) 
1 0.247 (0.081) 0.036 (0.021) 
Z 0.231 (0.077) 0.023 (0.018)* 
3 0.249 (0.082) 0.043 (0.023) 
4 0.204 (0.070) 0.031 (0.019)* 0.010 (0.016)* 
5 0.211 (0.072) 0.021 (0.014)* 0.038 (0.021) 
6 0.209 (0.072) 0.012 (0.016)* 0.043 (0.023) 
7 0.206 (0.071) —_ 0.020 (0.014)* 0.004 (0.016)* 0.039 (0.022) 
C or higher grade (dichotomized) 
1 0.643 (0.227) 0.032 (0.037)* 
2 0.688 (0.242) 0.062 (0.056)* 
3 0.622 (0.218) -- 
4 0.591 (0.218) 0.027 (0.035)* 0.063 (0.057)* 
5 0.543 (0.199) 0.022 (0.031)* -- 
6 0.561 (0.210) 0.078 (0.064)* -- 
7 -- 


0.505 (0.196) 


0.016 (0.029)* 0.079 (0.065)* 


Note. p values < .05 unless denoted with *. The corresponding standardized linear regression coefficients for grade 
earned are provided in Table 2, and the standardized logistic regression coefficients for dichotomized outcomes are 
provided in Table 4. HSGPA = high school grade point average. 
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Table A2. Lower and Upper Quartiles for Accuracy Rates (AR), Increase in Accuracy Rates (AAR), and Success Rates (SR) across Institutions for 
Single-Predictor Logistic Regression Course Grade Models 


Unweighted analyses Weighted analyses 
Predictors AR VANS SR AR AVN SR 


A grade 
1 ACTW 57.7-63.9 10.1-21.2 54.1-58.6 57.7-61.9 9.9-17.1 54.6—58.4 
Z ACTE 60.6-66.0 11.4-24.2 57.2-60.5 60.6—65.4 11.8-18.5 57.7-60.9 
6 HSGPA 61.9-65.4 10.4-19.6 55.3-61.1 60.6-64.2 11.9-18.0 52.3-60.9 
OO Borhighergradesiti‘“‘aléisS™S™S™S™S™SSSCS 
1 ACTW 64.1-71.8 0.1—2.4 65.0-72.1 65.0-79.7 0.0-0.4 65.4—79.7 
2 ACTE 64.1-71.4 0.1-1.7 65.0-—72.4 65.2-78.9 0.0—-0.6 65.7—-78.9 
3 HSGPA 67.5-76.1 1.2-5.3 68.6-77.6 67.1—78.2 0.7-2.9 68.4—78.7 


Note. Based on 16 out of the 28 participating institutions that had viable models (i.e., probability curves that crossed 0.50) for all three predictors. In the weighted 
analyses, institutions with larger numbers of students were assigned larger weights (i.e., number of students at institution divided by total number of students in 
analysis). ACTW = ACT writing score. ACTE = ACT English score. HSGPA = high school grade point average. Numbers shown in table are Q1 (25th percentile) — 
Q3 (75th percentile). 
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Table A3. Lower and Upper Quartiles for Accuracy Rates (AR), Increase in Accuracy Rates (AAR), and Success Rates (SR) across Institutions for 
Multiple-Predictor Logistic Regression Course Grade Models 


Unweighted analyses Weighted analyses 
Predictors AR VANS SR AR PAVAN SS SR 


A grade 
6 pore 65.1—76.4 17.0-51.6 57.9-64.2 64.8-76.3 19.7-50.9 58.0-65.2 
HSGPA : : : : ; : : : : E : . 
ACTW + : 
7 ACTE + 66.1—76.8 17.4-52.2 58.5-65.6 65.8-76.7 20.2-52.0 58.9-65.9 
HSGPA 
B or higher grade 
ACTE + 
6 HSGPA 69.5-79.6 0.9-4.3 71.1-81.0 70.8—82.2 0.3-2.7 72.3-82.7 
ACTW + 
7 ACTE + 70.0-—79.4 1.1-4.8 71.9-80.9 71.1-82.3 0.5-3.5 72.8-83.0 
HSGPA 


Note. Based on 24 (26) of the 28 participating institutions that had viable A (B or higher) grade models (i.e., probability curves that crossed 0.50) for the two 
models being compared in the table. In the weighted analyses, institutions with larger numbers of students were assigned larger weights (i.e., number of students 
at institution divided by total number of students in analysis). ACTW = ACT writing score. ACTE = ACT English score. HSGPA = high school grade point average. 
Numbers shown in table are Q1 (25th percentile) — Q3 (75th percentile). 


Justine Radunzel, PhD 


Justine Radunzel is a principal research scientist in Validity and Efficacy Research specializing in 
postsecondary outcomes research and validity evidence for the ACT test. 


Acknowledgements 


The author thanks Jeff Allen, Dongmei Li, and Emily Neff for their valuable feedback on earlier drafts. 


ACT 


